{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5wFF5JFyD2Ki"
      },
      "source": [
        "#### Copyright 2019 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Uf6NouXxDqGk"
      },
      "outputs": [],
      "source": [
        "# Copyright 2019 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ORy-KvWXGXBo"
      },
      "source": [
        "# Exploring the TF-Hub CORD-19 Swivel Embeddings\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/hub/tutorials/cord_19_embeddings_keras\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/cord_19_embeddings_keras.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/hub/blob/master/examples/colab/cord_19_embeddings_keras.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/hub/examples/colab/cord_19_embeddings_keras.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yI6Mh3-P0_Pk"
      },
      "source": [
        "The CORD-19 Swivel text embedding module from TF-Hub (https://tfhub.dev/tensorflow/cord-19/swivel-128d/3)\n",
        " was built to support researchers analyzing natural languages text related to COVID-19.\n",
        "These embeddings were trained on the titles, authors, abstracts, body texts, and\n",
        "reference titles of articles in the [CORD-19 dataset](https://pages.semanticscholar.org/coronavirus-research).\n",
        "\n",
        "In this colab we will:\n",
        "- Analyze semantically similar words in the embedding space\n",
        "- Train a classifier on the SciCite dataset using the CORD-19 embeddings\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "gVWOrccw0_Pl"
      },
      "source": [
        "## Setup\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Ym2nXOPuPV__"
      },
      "outputs": [],
      "source": [
        "import functools\n",
        "import itertools\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import seaborn as sns\n",
        "import pandas as pd\n",
        "\n",
        "import tensorflow as tf\n",
        "\n",
        "import tensorflow_datasets as tfds\n",
        "import tensorflow_hub as hub\n",
        "\n",
        "from tqdm import trange"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_VgRRf2I7tER"
      },
      "source": [
        "# Analyze the embeddings\n",
        "\n",
        "Let's start off by analyzing the embedding by calculating and plotting a correlation matrix between different terms. If the embedding learned to successfully capture the meaning of different words, the embedding vectors of semantically similar words should be close together. Let's take a look at some COVID-19 related terms."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "HNN_9bBKSLHU"
      },
      "outputs": [],
      "source": [
        "# Use the inner product between two embedding vectors as the similarity measure\n",
        "def plot_correlation(labels, features):\n",
        "  corr = np.inner(features, features)\n",
        "  corr /= np.max(corr)\n",
        "  sns.heatmap(corr, xticklabels=labels, yticklabels=labels)\n",
        "\n",
        "# Generate embeddings for some terms\n",
        "queries = [\n",
        "  # Related viruses\n",
        "  'coronavirus', 'SARS', 'MERS',\n",
        "  # Regions\n",
        "  'Italy', 'Spain', 'Europe',\n",
        "  # Symptoms\n",
        "  'cough', 'fever', 'throat'\n",
        "]\n",
        "\n",
        "module = hub.load('https://tfhub.dev/tensorflow/cord-19/swivel-128d/3')\n",
        "embeddings = module(queries)\n",
        "\n",
        "plot_correlation(queries, embeddings)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Bg-PGqtm8B7K"
      },
      "source": [
        "We can see that the embedding successfully captured the meaning of the different terms. Each word is similar to the other words of its cluster (i.e. \"coronavirus\" highly correlates with \"SARS\" and \"MERS\"), while they are different from terms of other clusters (i.e. the similarity between \"SARS\" and \"Spain\" is close to 0).\n",
        "\n",
        "Now let's see how we can use these embeddings to solve a specific task."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "idJ1jFmH7xMa"
      },
      "source": [
        "## SciCite: Citation Intent Classification\n",
        "\n",
        "This section shows how one can use the embedding for downstream tasks such as text classification. We'll use the [SciCite dataset](https://www.tensorflow.org/datasets/catalog/scicite) from TensorFlow Datasets to classify citation intents in academic papers. Given a sentence with a citation from an academic paper, classify whether the main intent of the citation is as background information, use of methods, or comparing results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Ghc-CzT8DDaZ"
      },
      "outputs": [],
      "source": [
        "builder = tfds.builder(name='scicite')\n",
        "builder.download_and_prepare()\n",
        "train_data, validation_data, test_data = builder.as_dataset(\n",
        "    split=('train', 'validation', 'test'),\n",
        "    as_supervised=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "CVjyBD0ZPh4Z"
      },
      "outputs": [],
      "source": [
        "#@title Let's take a look at a few labeled examples from the training set\n",
        "NUM_EXAMPLES =   10#@param {type:\"integer\"}\n",
        "\n",
        "TEXT_FEATURE_NAME = builder.info.supervised_keys[0]\n",
        "LABEL_NAME = builder.info.supervised_keys[1]\n",
        "\n",
        "def label2str(numeric_label):\n",
        "  m = builder.info.features[LABEL_NAME].names\n",
        "  return m[numeric_label]\n",
        "\n",
        "data = next(iter(train_data.batch(NUM_EXAMPLES)))\n",
        "\n",
        "\n",
        "pd.DataFrame({\n",
        "    TEXT_FEATURE_NAME: [ex.numpy().decode('utf8') for ex in data[0]],\n",
        "    LABEL_NAME: [label2str(x) for x in data[1]]\n",
        "})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "65s9UpYJ_1ct"
      },
      "source": [
        "## Training a citaton intent classifier\n",
        "\n",
        "We'll train a classifier on the [SciCite dataset](https://www.tensorflow.org/datasets/catalog/scicite) using Keras.  Let's build a model which use the CORD-19 embeddings with a classification layer on top."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "yZUclu8xBYlj"
      },
      "outputs": [],
      "source": [
        "#@title Hyperparameters { run: \"auto\" }\n",
        "\n",
        "EMBEDDING = 'https://tfhub.dev/tensorflow/cord-19/swivel-128d/3'  #@param {type: \"string\"}\n",
        "TRAINABLE_MODULE = False  #@param {type: \"boolean\"}\n",
        "\n",
        "hub_layer = hub.KerasLayer(EMBEDDING, input_shape=[], \n",
        "                           dtype=tf.string, trainable=TRAINABLE_MODULE)\n",
        "\n",
        "model = tf.keras.Sequential()\n",
        "model.add(hub_layer)\n",
        "model.add(tf.keras.layers.Dense(3))\n",
        "model.summary()\n",
        "model.compile(optimizer='adam',\n",
        "              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "              metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "weZKWK-pLBll"
      },
      "source": [
        "## Train and evaluate the model\n",
        "\n",
        "Let's train and evaluate the model to see the performance on the SciCite task"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "cO1FWkZW2WS9"
      },
      "outputs": [],
      "source": [
        "EPOCHS =   35#@param {type: \"integer\"}\n",
        "BATCH_SIZE = 32#@param {type: \"integer\"}\n",
        "\n",
        "history = model.fit(train_data.shuffle(10000).batch(BATCH_SIZE),\n",
        "                    epochs=EPOCHS,\n",
        "                    validation_data=validation_data.batch(BATCH_SIZE),\n",
        "                    verbose=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "2sKE7kEyLJQZ"
      },
      "outputs": [],
      "source": [
        "from matplotlib import pyplot as plt\n",
        "def display_training_curves(training, validation, title, subplot):\n",
        "  if subplot%10==1: # set up the subplots on the first call\n",
        "    plt.subplots(figsize=(10,10), facecolor='#F0F0F0')\n",
        "    plt.tight_layout()\n",
        "  ax = plt.subplot(subplot)\n",
        "  ax.set_facecolor('#F8F8F8')\n",
        "  ax.plot(training)\n",
        "  ax.plot(validation)\n",
        "  ax.set_title('model '+ title)\n",
        "  ax.set_ylabel(title)\n",
        "  ax.set_xlabel('epoch')\n",
        "  ax.legend(['train', 'valid.'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "nnQfxevhLKld"
      },
      "outputs": [],
      "source": [
        "display_training_curves(history.history['accuracy'], history.history['val_accuracy'], 'accuracy', 211)\n",
        "display_training_curves(history.history['loss'], history.history['val_loss'], 'loss', 212)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BjvtOw72Lpyw"
      },
      "source": [
        "## Evaluate the model\n",
        "\n",
        "And let's see how the model performs. Two values will be returned. Loss (a number which represents our error, lower values are better), and accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "y0ExC8D0LX8m"
      },
      "outputs": [],
      "source": [
        "results = model.evaluate(test_data.batch(512), verbose=2)\n",
        "\n",
        "for name, value in zip(model.metrics_names, results):\n",
        "  print('%s: %.3f' % (name, value))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dWp5OWeTL2EW"
      },
      "source": [
        "We can see that the loss quickly decreases while especially the accuracy rapidly increases. Let's plot some examples to check how the prediction relates to the true labels:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "VzHzAOaaOVC0"
      },
      "outputs": [],
      "source": [
        "prediction_dataset = next(iter(test_data.batch(20)))\n",
        "\n",
        "prediction_texts = [ex.numpy().decode('utf8') for ex in prediction_dataset[0]]\n",
        "prediction_labels = [label2str(x) for x in prediction_dataset[1]]\n",
        "\n",
        "predictions = [label2str(x) for x in model.predict_classes(prediction_texts)]\n",
        "\n",
        "\n",
        "pd.DataFrame({\n",
        "    TEXT_FEATURE_NAME: prediction_texts,\n",
        "    LABEL_NAME: prediction_labels,\n",
        "    'prediction': predictions\n",
        "})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OSGcrkE069_Q"
      },
      "source": [
        "We can see that for this random sample, the model predicts the correct label most of the times, indicating that it can embed scientific sentences pretty well."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oLE0kCfO5CIA"
      },
      "source": [
        "# What's next?\n",
        "\n",
        "Now that you've gotten to know a bit more about the CORD-19 Swivel embeddings from TF-Hub, we encourage you to participate in the CORD-19 Kaggle competition to contribute to gaining scientific insights from COVID-19 related academic texts.\n",
        "\n",
        "* Participate in the [CORD-19 Kaggle Challenge](https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge)\n",
        "* Learn more about the [COVID-19 Open Research Dataset (CORD-19)](https://pages.semanticscholar.org/coronavirus-research)\n",
        "* See documentation and more about the TF-Hub embeddings at https://tfhub.dev/tensorflow/cord-19/swivel-128d/3\n",
        "* Explore the CORD-19 embedding space with the [TensorFlow Embedding Projector](http://projector.tensorflow.org/?config=https://storage.googleapis.com/tfhub-examples/tensorflow/cord-19/swivel-128d/3/tensorboard/projector_config.json)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "Exploring the TF-Hub CORD-19 Swivel Embeddings",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
